Cognitive Principles of Graphic Displays

Graphic displays, such as maps, that portray visible things are ancient whereas graphic displays, such as graphs and diagrams, that portray things that are inherently not visible, are relatively modern inventions. They serve a variety of functions, such as providing models, attracting attention, supporting memory, facilitating inference and discovery. Graphic displays use space to convey meaning in ways that are cognitively natural, as suggested by historical and developmental examples. Typically, icons are used to convey elements, based on likenesses and "figures of depiction" and spatial relations are used to convey other relations, based on proximity. Cognitive Principles of Graphic Displays Graphics are one of the oldest and newest form of communication. Long before there was written language, there were pictures, of myriad varieties. A few of the multitude of cave paintings, petroglyphs, bone incisions, clay impressions, stone carvings, and wood markings that people fabricated and used remain from ancient cultures. Some of these prealphabetic depictions probably had religious significance, but many were undoubtedly used to communicate, to keep track of events in time, to note ownership and transactions of ownership, to map places, to record songs and sayings, and to transmit messages (e. g., Coulmas 1989; De Frances 1989; Gelb 1963; Mallery 1893/1972; Schmandt-Besserat 1992). As such, they served as permanent records of history, commemorations of cultural past. Because pictures represent meaning more directly than alphabetic written languages, we can guess at their meanings today. In rare cases, we have the benefit of contemporaneous translations, Mallery, for example, was able to speak with many still using pictographic communication as he collected vast numbers of petroglyphs, birch bark markings, and the like from native Americans (1893/1972). In many places in the world, the use of pictures to communicate developed into complete written languages. All such languages invented ways to represent concepts that are difficult to depict, such as abstract meanings and proper names. Some originally pictoric written languages transformed to using written marks to represent the sound of spoken language rather than using marks to represent meaning directly. As pictures evolved into written languages, their transparency necessarily disappeared. Characters representing abstract concepts were devised and characters representing concrete concepts became schematized and conventionalized. Later, the invention and spread of the alphabet, and then the invention of the movable type printing press decreased reliance on pictures for communication. With the increasing ease of reproducing written language and the spread of literacy, pictures became decorative rather than communicative. Now, pictures, depictions, and visualizations are on the rise again. As with the proliferation of written language, this is partly due to technologies for reproducing and transmitting pictures. And as with the proliferation of written language, some of the expansion of pictures is due to intellectual insights. For this, the basic insight is using depictions to represent abstract meaning by means of of visual and spatial metaphors and figures of depiction. Although depictions have long been used to convey concrete ideas, their use in conveying abstract ideas is more recent. Early depictions for the most part portrayed things that were inherently visualizable, such as objects or environments, in pictographs, maps, or architectural plans. Many contemporary depictions are visualizations of things that are not inherently visualizable, such as temporal, monetary, causal, or social relations. Graphs are perhaps the most prevalent example of depictions of abstract concepts, and were invented as recently as the late eighteenth century (e. g., Beniger and Robyn 1978; Carswell and Wickens 1988; Tufte1983), although they probably had their roots in mathematical notation, especially Cartesian coordinate systems. Two Europeans, Playfair in England and Lambert in Switzerland, are credited with being the first to promulgate their use, for the most part to portray economic and political data. Although those early graphs, X-Y plots with time as one of the variables, are still the most common type of graph in scientific journals (Cleveland 1984), varieties of graphs, graphics, and visualizations abound, with new ones appearing all the time. Bar graphs and pie charts are common for representing quantitative data, with flow charts, trees, and networks widely used for qualitative data. Icons appear in airports, train stations, and highways all over the world, and menus of icons on information 116 From: AAAI Technical Report FS-97-03. Compilation copyright © 1997, AAAI (www.aaai.org). All rights reserved. highways over the world. Many are used to portray concepts that are difficult to visualize. The choices of icons and graphic displays are usually not accidental or abitrary. Many have been invented and reinvented by adults and children across cultures and time. Many have analogs in language and in gesture. Many are rooted in natural cognitive correspondences, "figures of depictions," and spatial metaphors, and have parallels in Gestalt principles of perceptual organization. In this paper, I present an analysis of graphic displays based on their functions and on their structure. The" evidence I will bring to bear is eclectic and unconventional, drawing from examinations of historical graphic inventions, children’s graphic inventions, and language. Others have taken a broad view of graphics from other perspectives. Bertin (1983) put forth a comprehensive semiotic analysis of the functions of graphics and the processes used to interpret them that established the field and defined the issues. According to Bertin, the functions of graphs are to record, communicate, and process information, and the goal of a good graphic is simplification to those ends. Ittelson (1996) has pointed differences in processing of "markings," deliberate, twodimensional inscriptions on surfaces of objects and other visual stimuli. Winn (1987) has discussed how information is conveyed in charts, diagrams, and graphs. Larkin and Simon (1987) have examined the differences between sentential and diagrammatic external representations, pointing to the advantages of diagrammtic ones for tasks where spatial proximity conveys useful information. Stenning and Oberlander (1995) have analyzed the advantages and disadvantages of diagrammtic and sentential representations in drawing inferences. They argue that diagrams allow expression of some abstractions, much like natural language, but are not as expressive as sentential logics. Cleveland (1984, 1985) has examined the psychophysical advantages and disadvantages of using different graphic elements, position, angle, length, slope, and more, for efficiency in extracting different kinds of information from displays of quantitative data. He and his collaborators have produced convincing cases where conventional data displays can be easily misconstrued by human users. Tufte (1983, 1990, 1997) has exhorted graphic designers to refrain from "chart junk," extraneous marks that convey no additional information, adopting by contrast a minimalist view. Wainer (1984, 1992) has gathered a set of useful prescriptions and insightful examples for graph construction, drawing on work in semiotics, design, and information processing. Kosslyn (1985, 1994), using principles adopted from visual information processing and Goodman’s (1978) analysis symbol systems, has developed a set of prescriptives for graphic design, based on an analysis of the syntax, semantics, and pragmatics underlying graphs. Pinker (1990) provides an analysis of information extraction from graphics that separates processes involved in constructing a visual description of the physical aspects of the graph from those involved in constructing a graph schema of the mapping of the physical aspects to mathematical scales. Carswell and Wickens (Carswell 1992; Carswell Wickens 1988; t990) have demonstrated effects of perceptual analysis of integrality on graph comprehension, and others have shown biases in interpretation or memory dependent on graphic displays (Gattis & Holyoak 1996; Levy, et al. 1996; Schiano & Tversky 1992; Shah & Carpenter 1995; Spence & Lewandowsky 1991; Tversky & Schiano 1989). Some Functions of Graphic Displays Despite their variability of form and context, a number of cognitive principles underliegraphic displays. These are evident in the many functions they serve as well as in the way information is conveyed in them. Some of their many overlapping functions are reviewed below. As with functions, goals, and constraints on other aspects of human behavior, so the functions of graphic displays are sometimes at odds with each other. Attract attention and interest. One prevelant function of graphic displays is to attract attention and interest. A related function is aesthetic; graphics may be pleasing or shocking or repulsive or calming or funny. Models of actual and theoretical worlds. Maps, architectural drawings, molecules, circuit diagrams, organizational charts, flow diagrams are just some of the myriad examples of diagrams serving as models of worlds and the things in them. This function includes both the ancient examples of maps and the contemporary examples of organizational charts and flow diagrams. Note that these are models, and not strictly shrunken or expanded worlds. Effective diagrams omit many features of the modeled world and distort others, and even add features that are not in the modeled world. Maps, for example, are not drawn strictly to scale. Roadmaps exaggerate the sizes of highways and streets so that they can be seen and used. Maps introduce symbolic elements, for railroads, ocean depth, towns, and more, that require a key and/or convention to interpret. The essence of creating an effective external representation is to abstract those features that are essential and to eliminate those that are not, that only serve as clutter. Of course, this is not as simple as it sounds, partly because it is difficult to anticipate all the uses an external representation will have, partly because successful communication rests on redundancy. Current trends in computer graphics go against the maxim of abstracting the essentials. The aim of at least some areas of computer graphics seem to be creating as much detail and realism as possible. Record information. An ancient function of graphics is to provide records. For example, tallies developed to keep track of property, beginning with a simple one-mark one-